Skip to content

fix(ep/v2): AsyncLL fused-receive deadlock under two streams; early notification by default on gfx950 - #875

Merged
yangyuhuiling merged 4 commits into
mainfrom
xiangch/asyncll-gfx950-early-default
Nov 5, 2026
Merged

yangyuhuiling merged 4 commits into
mainfrom
xiangch/asyncll-gfx950-early-default

Conversation

@yangyuhuiling

Copy link
Copy Markdown
Contributor

Summary

Two changes to the v2 AsyncLL family (follow-up to #768):

  1. Fix a cross-rank deadlock with MORI_ASYNCLL_FUSED_RECV=1 and two streams. The fused receive grid is now capped at half the device by default.
  2. Default to early notification on gfx950 (MI355X). It beats the legacy sequence at every token count measured.

1. Fused receive deadlock (reported from sglang two-batch overlap)

A fused receive copy (asyncLlMode 2) waits for its peers inside every CTA, so its whole grid stays resident for the length of the wait. Two micro-batches on two streams put two fused copies on the device. When those span the device, there is no room left for the send pass the other stream must run before any peer's wait can end, and the ranks deadlock.

Two configurations are unaffected:

  • Early notification without fusion: it waits in 8 waves.
  • A single stream: sends precede receives in stream order.

Measured occupancy confirms this. A fused recvcopy runs 4 CTAs/CU on gfx950, so two uncapped grids of 256 occupy half the device's CTA slots for the duration of the wait.

Fix:

  • MakeEpInterNodeCfg resolves asyncLlFusedRecvCtas = mpCount / 2. MORI_ASYNCLL_FUSED_RECV_CTAS overrides it, as a tuning knob only.
  • The AsyncLL geometry clamps the two mode-2 receive copies to that value. RecvCopy strides over its tokens, so a narrower grid does the same work in fewer CTAs.
  • The ceiling is a Cfg field, so Geometry() stays a pure function of the Cfg.

Validation on MI355X, 8 GPUs, with a dual-stream TBO reproducer and CUs restricted via HSA_CU_MASK:

uncapped default ceiling (grid 128)
64 CUs deadlocks 3/3 complete
128 CUs deadlocks, or stalls to ~124 ms/iter 3/3 complete (8.5 ms/iter)

test_asyncll (sdma) passes with fused receive and the default ceiling.

Cost on the full device, median of 3, ms/iter:

early fused uncapped fused, ceiling 128
M=4096, 1 stream 3.386 3.260 3.418
M=4096, 2 streams 3.028 3.042 3.056
M=512, 1 stream 0.797 0.751 0.765
M=512, 2 streams 0.670 0.721 0.691

Fusion gains nothing over early notification with two streams. The docs now say so and recommend early notification alone for that case. The ceiling makes the deadlock unlikely but cannot rule it out if another stream fills most of the device.

2. Early notification by default on gfx950

EP8, hidden 7168, top-6, bf16, all ranks loaded, full device, median of 3 interleaved runs, ms/iter:

legacy early Δ
M=4096, 1 stream 3.399 3.337 −1.8%
M=4096, 2 streams 3.369 3.014 −10.5%
M=512, 1 stream 0.813 0.790 −2.8%
M=512, 2 streams 0.722 0.681 −5.7%
M=128, 1 stream 0.570 0.542 −4.9%
M=128, 2 streams 0.501 0.485 −3.2%
  • MORI_ASYNCLL_EARLY_NOTIFY=0 restores legacy.
  • gfx942 is unmeasured and keeps legacy.
  • The CI AsyncLL step now runs the sdma leg three times: default (early), legacy, and fused. Neither non-default protocol loses coverage.

Tests

  • test_jit_core: new AsyncLLMode.FusedReceiveGridIsCappedAtHalfTheDeviceByDefault. It checks the default and explicit ceilings, alignment to worldSize, that the ceiling never widens the grid, and that invalid or misplaced ceilings are rejected.
  • test_asyncll_options: the ceiling env var reaches only mode 2 and rejects bad values; gfx950 defaults to early; gfx942 still requires opt-in.
  • These and test_asyncll (sdma, early and fused) passed on MI355X. That run used the fix on top of the feat(ep/v2): AsyncLL dispatch/combine on cco, with cross-node and gfx125x support #768 branch; this PR rebases it onto current main.

🤖 Generated with Claude Code

xiangch and others added 2 commits October 30, 2026 07:36
A fused receive copy (MORI_ASYNCLL_FUSED_RECV=1) waits for its peers
inside every CTA, so its whole grid stays resident for the length of the
wait. With two micro-batches on two streams (sglang two-batch overlap),
two fused copies spanning the device can leave no room for the send pass
the other stream must run before any peer's wait can end, and the ranks
deadlock. Early notification without fusion waits in eight waves and is
unaffected; so is a single stream, where sends precede receives.

Resolve a grid ceiling in MakeEpInterNodeCfg -- mpCount / 2 unless
MORI_ASYNCLL_FUSED_RECV_CTAS overrides it -- and apply it in the AsyncLL
geometry to the two receive copies of mode 2 only. RecvCopy strides over
its tokens, so a narrower grid does the same work in fewer CTAs. The
ceiling travels in the Cfg, so Geometry() stays a pure function of it.

Measured on MI355X (8 GPUs, dual-stream TBO repro, CUs restricted with
HSA_CU_MASK): uncapped, the repro deadlocks or stalls to ~124 ms/iter;
with the default ceiling 6/6 runs complete at 64 and 128 CUs.
test_asyncll passes with fusion and the default ceiling. Full device,
median of 3, ms/iter:

                     early   fused uncapped   fused (cap 128)
  M=4096 1 stream    3.386        3.260            3.418
  M=4096 2 streams   3.028        3.042            3.056
  M=512  1 stream    0.797        0.751            0.765
  M=512  2 streams   0.670        0.721            0.691

Fusion gains nothing over early notification on two streams; the docs
now say so and recommend early notification alone for that case.

Co-Authored-By: Claude <noreply@anthropic.com>
Early notification (asyncLlMode 1) posts each payload's ready word on the
same SDMA queue behind the payload, so the receive phase no longer waits
for a separate transfer pass. It was on by default only for gfx1250.

On MI355X it beats the legacy sequence at every token count measured.
EP8, hidden 7168, top-6, bf16, all ranks loaded, full device, median of
3 interleaved runs, ms/iter:

                 legacy   early
  M=4096 1 str    3.399   3.337   -1.8%
  M=4096 2 str    3.369   3.014  -10.5%
  M=512  1 str    0.813   0.790   -2.8%
  M=512  2 str    0.722   0.681   -5.7%
  M=128  1 str    0.570   0.542   -4.9%
  M=128  2 str    0.501   0.485   -3.2%

The largest gain is with two overlapping streams (two-batch overlap),
the case AsyncLL exists for. MORI_ASYNCLL_EARLY_NOTIFY=0 restores the
legacy sequence. gfx942 is unmeasured and keeps legacy.

Since early is now what CI's gfx950 runner gets by default, the sdma leg
of the AsyncLL CI step runs three times: default (early), legacy, and
fused receive, so neither of the other two loses coverage.

Co-Authored-By: Claude <noreply@anthropic.com>

@jhchouuu jhchouuu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested 874fa999 on MI355X at EP4 and EP8 (hidden 7168, top-6, bf16, sdma transport), with this PR and main each built from source. I used a two-stream reproducer: two ops on one communicator, one micro-batch per stream, dispatch_send → dispatch_recv → combine_send → combine_recv. I restricted CUs with HSA_CU_MASK where noted. In the "skewed" order, odd ranks enqueue micro-batch 1 first.

Early notification by default looks right. It passes test_asyncll.py (early default, legacy and fused) at both 4 and 8 ranks, and test_asyncll_options.py (7 passed). Two streams, full device, median ms/iter:

M EP4 main (legacy) EP4 PR default (early) EP4 PR fused EP8 main (legacy) EP8 PR default (early) EP8 PR fused
2048 2.04 1.82 1.86 1.68 1.32 1.36
512 5.62 0.51 0.51 0.48 0.40 0.41

Early is faster everywhere. At EP4, M=512, legacy also stalls with two streams (median 5.6 ms, max 60 ms), and early avoids that, which supports the default switch.

The fused-receive ceiling helps, but in my reproducer it does not remove the deadlock:

fused receive, M=2048 main, EP4 main, EP8 this PR, EP4 this PR, EP8
128 CUs, skewed order completes, median 37 ms, one 12.4 s stall deadlock 2.2 ms 1.8 ms
64 CUs, skewed order deadlock deadlock deadlock deadlock
64 CUs, same order on every rank — — 2.5 ms —
64 CUs, skewed, MORI_ASYNCLL_FUSED_RECV_CTAS=64 — — completes, but median 40.8 ms and an 18 s stall —
64 CUs, skewed, early (no fusion) — — 2.4 ms 2.3 ms
64 CUs, skewed, legacy 2.5 ms 2.3 ms — —

"Deadlock" means no progress before a 240 s timeout, with no error.

I think the reason is that mpCount is the device's full count, while HSA_CU_MASK, or anything else holding CUs, only shrinks what is actually available. With 64 CUs the two capped grids still fill every CTA slot.

The description says the ceiling "cannot rule it out", and the default is early notification without fusion, so this is not blocking. It would help if the docs said fused receive should not be combined with two streams when the device may be shared. Better still would be to size the ceiling from the CUs the process can actually use rather than from multi_processor_count.

…eaves

The fused receive grid ceiling defaulted to mpCount / 2, and mpCount is
the device's full CU count even when HSA_CU_MASK gives the process a
fraction of it. Reported in review: at 64 of 256 CUs, two 128-CTA fused
grids still fill every CTA slot, and with micro-batches on two streams
enqueued in opposite orders on neighbouring ranks the ranks deadlock just
as without a ceiling.

When no MORI_ASYNCLL_FUSED_RECV_CTAS override is given and HSA_CU_MASK
restricts this device, the host now passes half of the CUs the mask
leaves as the ceiling. A mask it cannot read leaves the whole-device
default in place rather than failing the op. CUs held by another stream,
kernel or process at run time cannot be seen this way, so the docs now
say plainly not to combine receive fusion with two streams, and that
early notification -- the gfx950 default -- is the dual-stream choice.

Co-Authored-By: Claude <noreply@anthropic.com>
@yangyuhuiling

Copy link
Copy Markdown
Contributor Author

@jhchouuu Thanks for the thorough test. You're right about the cause, and both suggestions are in bbe293c.

Ceiling sized from the CUs the process can use. When MORI_ASYNCLL_FUSED_RECV_CTAS is not set and HSA_CU_MASK restricts this device, the host now passes half of the CUs the mask leaves, instead of letting the native default use half of multi_processor_count. At HSA_CU_MASK=0-7:0-63 that gives 32 instead of 128. Details:

  • The parser accepts ROCr's <gpus>:<cus>[;...] form with comma lists and a-b ranges.
  • A mask it can't read, or one that doesn't restrict this GPU, keeps the whole-device default instead of failing the op.
  • An explicit MORI_ASYNCLL_FUSED_RECV_CTAS still wins.
  • GPU indices are taken in ROCr enumeration order, which matches torch's unless HIP_VISIBLE_DEVICES reorders devices.

Docs. The MORI_ASYNCLL_FUSED_RECV_CTAS entry now says the default is half of what the process may use. A separate paragraph says not to combine receive fusion with micro-batches on two streams, especially when the device may be shared, and that early notification (the gfx950 default) is the dual-stream choice. The ceiling still can't see CUs another stream, kernel or process holds at run time, and the docs say so.

Your case, re-run on 8× MI355X (ROCm 10 image): EP8, M=2048, bf16, fused receive, two streams, odd ranks enqueue micro-batch 1 first, 60 iterations per run.

874fa999 bbe293c4
64 CUs deadlock 3/3 (grid 128) completes 3/3, 2.37–2.41 ms (grid 32)
128 CUs completes 2/2, 1.93–1.95 ms (grid 128) completes 2/2, 2.00–2.02 ms (grid 64)
full device completes 2/2, 1.86 ms (grid 128) completes 2/2, 1.86–1.88 ms (grid 128)
64 CUs, early only (no fusion) — completes, 2.21 ms

At 128 CUs the smaller grid costs about 4%, the price of the extra headroom. Without a mask nothing changes.

test_asyncll_options.py passes (8 tests, 21 subtests, including the new mask cases), and so does test_asyncll.py with fused receive on the new head.

One unrelated note, since you are testing two-stream setups: on the rocm/sgl-dev:v0.5.23-rocm720 image, two streams running GEMMs at the same time hang without mori involved at all. The stuck kernel is hipBLASLt's Stream-K GEMM. That can look like an AsyncLL deadlock, so a ROCm 10 image is the safer base for these tests.

@jhchouuu jhchouuu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-tested bbe293c4 on 8x MI355X (ROCm 7.14, FlyDSL 0.4.0) with the same two-stream reproducer. The ceiling now follows the mask, and the deadlock is gone.

EP8, M=2048, bf16, two streams, odd ranks enqueue micro-batch 1 first, median ms:

874fa999 bbe293c4
fused, 64 CUs deadlock 2.45 (3/3)
fused, 128 CUs 1.83 1.86
fused, full device 1.36 1.37
early only, 64 CUs — 2.27

At EP4 with 64 CUs and the same skewed order, it now completes as well (2.67 ms, 2/2); it deadlocked there before. test_asyncll.py with fused receive (8 ranks) and test_asyncll_options.py (8 tests, 21 subtests) pass. The docs paragraph reads well. LGTM.

Before merging: the branch now conflicts with main in docs/api/communication.rst.

…0-early-default

# Conflicts:
#	docs/api/communication.rst
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants